Skip to content

Add Boson realtime plugin#6340

Open
helin-mktech wants to merge 7 commits into
livekit:mainfrom
helin-mktech:feat/boson-realtime-plugin
Open

Add Boson realtime plugin#6340
helin-mktech wants to merge 7 commits into
livekit:mainfrom
helin-mktech:feat/boson-realtime-plugin

Conversation

@helin-mktech

@helin-mktech helin-mktech commented Jul 7, 2026

Copy link
Copy Markdown

Summary

Adds a Boson realtime plugin for LiveKit Agents.

  • Adds livekit-plugins-boson with boson.realtime.RealtimeModel.
  • Translates LiveKit realtime session operations to Boson's OpenAI-compatible realtime WebSocket protocol, including session updates, audio input, text/audio response streams, user transcription events, interruptions, errors, and function calls.
  • Registers the plugin package in the workspace and includes README documentation plus focused realtime regression coverage.

Notes

  • The plugin supports one output modality per session: ["audio"] by default, or ["text"] for text-only responses.
  • Server-side turn detection defaults to server VAD and can be disabled with turn_detection=None.
  • User transcription events are enabled when input_audio_transcription contains a non-empty model.

@helin-mktech helin-mktech requested a review from a team as a code owner July 7, 2026 09:04
@CLAassistant

CLAassistant commented Jul 7, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

helin-mktech and others added 2 commits July 7, 2026 17:20
…ntime

Addresses the review finding that a single WebSocket failure was
terminal. Instead of extending the forked connection machinery, drop it
(~400 lines) and inherit the parent OpenAI plugin's
_main_task/_run_ws/_reconnect loop, xAI-style; Boson-specific behavior
now lives in small overrides and hooks:

- Reconnection follows the server's documented contract (the voice-chat
  server keeps no state across connections): the base _reconnect re-sends
  the full session config and replays the local chat-context mirror via
  conversation.item.create, with _create_session_update_event /
  _create_tools_update_event / _create_update_chat_ctx_events overridden
  to emit Boson-shaped payloads. Retry count/backoff come from
  conn_options, and a session_reconnected event is emitted on resume.
- _run_ws wraps the base loop only to reclassify closes a reconnect
  cannot fix as non-retryable: close code 3000 (invalid API key) and
  server-announced session ends (session.idle_timeout,
  session.max_duration_reached), which must not be resurrected.
- Boson-only server events are handled off the base's
  openai_server_event_received hook instead of a forked dispatch;
  _main_task fails pending response futures immediately on terminal
  failure instead of leaving them to their own timeouts.
- Stop advertising auto_tool_reply_generation: the server treats
  conversation.item.create as a pure insert now, so the framework must
  send response.create after tool outputs — previously tool calls
  completed client-side but no reply audio ever came back.
- Treat conversation.item.added for a known id as an in-place update:
  the server merges consecutive same-role turns into one item and
  re-emits added with cumulative content, which the base insert rejected
  ("already exists"), leaving the mirror with stale content.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

The _handle_response_created override closed the current generation and
recorded _current_response_id before the base handler could check
_discarded_event_ids. A response that timed out or was interrupted
before the server created it would therefore cut off the streams of a
newer in-progress generation, and a subsequent interrupt() would cancel
the stale response id instead of the active one.

Check the discard set first and let the base handler cancel/discard the
stale response without touching the in-progress generation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

helin-mktech and others added 2 commits July 8, 2026 15:14
The previous fix stopped closing the active generation when a discarded
response's response.created arrived, but the base handler still parked a
_DiscardedGeneration marker in the generation slot, so every subsequent
delta of the legitimate in-progress response was silently dropped — and
plain save/restore would only defer the damage, because the stale
response's response.done (which follows the cancel almost immediately)
would close whatever generation sits in the slot: neither the slot nor
_handle_response_done is response-id aware.

When a legitimate generation is streaming, restore it after the base
discards the stale response AND remember the stale response id;
_handle_response_done ignores terminal events for remembered ids instead
of closing the active generation or clearing _current_response_id. With
no generation in progress (the common serial case) the base discard
marker is kept so the stale response's trailing events are still eaten.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… ids

The server now stores client-supplied item ids and echoes them on
conversation.item.added (duplicate ids are rejected instead of merged), so
item ids work as the foreign key the base machinery expects. Drop the
append-only update_chat_ctx and its manual seen-id ledger in favor of the
base implementation: diff against the remote mirror, create/delete sync,
and echo-correlated futures.

_create_update_chat_ctx_events shrinks to a thin adapter: it feeds the base
diff a text-only mirror of the context (audio is represented by its
transcript; the GA converter must never see audio frames, whose conversion
can raise on empty buffers) and rebuilds each create with the Boson item
shape, mapping the unsupported "root" previous_item_id sentinel to a tail
append. A minimal extra="allow" item model keeps the base isinstance/echo
plumbing working without fighting the GA content-type literals.

With items addressable, mutable_chat_context returns to the base default
(True): interrupted never-played messages are now cleaned up server-side
and realtime sessions can be reused across agent handoffs.

Also trim nonstandard surface while here: drop the RealtimeModel.aclose
override that closed sessions (session lifecycle belongs to the framework)
and the _pending_response_futures test-compat alias, remove the unused
NUM_CHANNELS re-export and single-use helpers, warn once instead of
per-frame in push_video, and reword comments to describe wire behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

A response id enters _boson_discarded_response_ids when its stale
response.created arrives while another generation is streaming, and leaves
it on the matching response.done. If the connection drops in between, that
done never arrives, so the entry would outlive its connection. Clear the
set on session_reconnected alongside the other per-connection state,
matching how the base reconnect clears _discarded_event_ids.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants